Motion detection method and system

ABSTRACT

Techniques for capturing motion of a user are described. A plurality of sensor modules corresponding to a set of designated body parts (e.g., arms or legs) of the user generate sensing signals when the user performs a pose. Sensing data including accelerometers and/or gyroscopes data produced from the sensing signals is analyzed to derive various moves made by the user, where an human avatar animation is provided and displayed based on motion attributes derived from the sensing data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/533,053, entitled “Techniques for establishing biomechanical modelthrough motion capture”, filed on Nov. 22, 2021 and now U.S. Pat. No.11,551,396, which is a continuation of U.S. application Ser. No.16/687,677, entitled “Motion management via conductive threads embeddedin clothing material”, filed on Nov. 18, 2019 and now U.S. Pat. No.11,182,946, which is a continuation-in-part of U.S. application Ser. No.16/423,130, entitled “System and method for capturing and analyzingmotions to be shared”, filed on May 27, 2019 and now U.S. Pat. No.10,672,173, which is a continuation of U.S. application Ser. No.16/219,727, entitled “System and method for capturing and analyzingmotions to render a human avatar animation”, filed on Dec. 13, 2018 andnow U.S. Pat. No. 10,304,230, which is a continuation of U.S.application Ser. No. 15/271,205, entitled “System and method forcapturing and analyzing motions”, filed on Sep. 20, 2016 and now U.S.Pat. No. 10,157,488, which claims the priority of U.S. Prov. App. Ser.No. 62/221,502, entitled “System and method for capturing and analyzingcomplex motions”, filed on Sep. 21, 2016.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention is generally related to motion controls and moreparticularly related to methods and systems for motion controls based onartificial intelligence and providing instructions to a user to mimicmotions performed by an instructor. The present invention is alsoparticularly related to sensorized electronic garments (eGarments) tofacilitate the capture of motions performed by a wearer and varioustechniques to derive motions of the wearer based on data from sensorsembedded in eGarments.

Description of the Related Art

Current wearable devices in the market are limited to tracking simplerepetitive activities like walking, running, and swimming. They countsimple statistics like how many/much steps/strokes, calories, and heartrates per period. They are prone to inaccuracies and less beneficial ifused for complex sports like yoga, fitness, golf, and tennis. Someefforts are attempting to address the problem by adding a 9-axisinertial sensor (e.g., 3-axis gyroscope+3-axis accelerometer+3-axismagnetometer) in the equipment (e.g., racquet, club, bat), or providing1 or 2 sensors in a user (e.g., placed near a wrist, ankle, or ear).What users get however are still limited to “after-the-fact” statistics(e.g., repetition count, speed, or cadence), some numbers that are lessuseful to tell what the user did or did not do correctly, how to improvethe technique or reduce injury risk.

Thus there is a great need for methodologies or systems that are capableof motion management without confining the motions performed by a user,providing real-time feedback and authoritative coaching and/orinstruction.

SUMMARY OF THE INVENTION

This section is for the purpose of summarizing some aspects of thepresent invention and to briefly introduce some preferred embodiments.Simplifications or omissions may be made to avoid obscuring the purposeof the section. Such simplifications or omissions are not intended tolimit the scope of the present invention.

In general, the present invention is related to techniques for motioncontrols based on artificial intelligence. According to one aspect ofthe present invention, instructions are provided based on motionsperformed by a user in reference to motions performed by an instructor.Various parameters or attributes about the motions by the user areanalyzed, derived and compared with stored parameters pertaining tomotions performed by an authoritative person (i.e., instructor). Ananimation based on the user or an avatar representing the user isrendered per the motion parameters. Various techniques or algorithms aredesigned to provide different perspective views of the motions by theuser and the instructor and compare the motions or poses by the user andthe instructor.

According to another aspect of the present invention, an article ofclothing is uniquely designed to capture motions by a wearer (user),where a plurality of sensors or sensor modules are respectively attachedto or embedded in different parts of the clothing. Depending on how thesensors or sensor modules operate, specially designed conductive wiresare provided within the clothing to provide a communication mediumbetween and/or among the sensors or sensor modules. Depending onimplementation, the sensors may communicate with a designated sensorwirelessly or via the medium while the designated sensor communicateswirelessly with an external device. When the clothing is worn by awearer or user, these embedded sensors facilitate the capture of motionsperformed by the wearer without confining the wearer to a limited numberof motions or poses.

According to another aspect of the present invention, some or all of thesensors are designated to react to certain actions from a user togenerate a command signal when the user taps on a specific part ofhis/her body, where the command signal causes a system (e.g., anexternal device) to respond to the command signal by, for example,changing or repeating a perspective view of motion or pose beingperformed by an instructor.

According to still another aspect of the present invention, a library ofactivities (e.g., tennis or Yoga) is provided in a computing device thatallows the user to choose one therefrom to exercise. The library alsoprovides a group of instructors for the chosen activity. A video of achosen instructor performing the activity is displayed after one of theinstructors is chosen so that the user may follow the instructor toperform the activity. The video is modified or enhanced to include anavatar representing the user next to the representation of theinstructor so that a comparison of two performing the same activity canbe provided.

According to still another aspect of the present invention, a display isprovided based on the motions by a user. The display includes at leasttwo avatars representing the user and an instructor, where variousperspective views of the two avatars can be provided, errors in motionor pose differences can be highlighted, corrected when the user changeshis/her motions, and progressive scores of the comparisons can also beprovided.

According to still another aspect of the present invention, aperspective view of comparisons between the user and the instructorperforming the same activity is automatically determined to allow theuser to correct his/her move so as to reduce or minimize the differencesin their moves. The angle of the perspective video may be determinedbased on a set of procedure specified by the instructor, a possiblecause of error by the user in his/her move and a move needed by a bodypart and etc.

According to yet another aspect of the present invention, a perspectiveview is automatically provided when the errors are beyond a threshold,where the perspective view is determined based on a largest differencebetween two corresponding parts in the two avatars and shows thedifference between the two corresponding parts.

The present invention may be implemented as a method, a system, anapparatus, a part of a system, and an article of clothing. Differentimplementations yield different benefits, advantages and objectives.According to one embodiment, the present invention is a motionmanagement system comprising an article of clothing having a layer ofmaterial, and a plurality of sensor modules respectively attached todesignated locations on the inner side of the clothing. Each of thesensor modules corresponds to a designated body part of a wearer of theclothing. At least one of the sensor modules is designated as a hubmodule and the rest of the sensor modules are designated as satellitemodules. Each of the satellite modules includes a microcontroller, atleast an inertial sensor and a transceiver for intercommunication withthe hub module. The hub module includes a microcontroller, at least aninertial sensor and a transceiver for intercommunication with thesatellite modules and another transceiver for communicating with anexternal computing device.

According to another embodiment, the present invention is a motionmanagement system comprising an article of clothing having a layer ofmaterial, and a plurality of sensor modules respectively attached todesignated locations on the layer of material, each of the sensormodules corresponding to a designated body part of a wearer of theclothing, wherein one of the sensor modules is designated as a hubmodule and the rest of the sensor modules are designated as satellitemodules, each of the satellite modules includes an inertial sensor, thehub module includes a microcontroller, at least an inertial sensor andan interface to receive sensing signals from inertial sensors in thesatellite modules via respective conductive threads embedded in thelayer of material and a transceiver for communicating with an externalcomputing device.

According to still another embodiment, the present invention is a methodfor motion management. The method comprising receiving in an externaldevice sensing data from a plurality of sensor modules, derive from thesensing data a set of attributes pertaining to motions performed by auser, a player or a wearer of an article of specially designed clothing,and rendering a display showing an avatar representing the wearer,wherein the avatar is animated as the wearer moves. The display may alsobe rendered to show a representation of another person selected from alist of instructors. The representation is animated based on storeddata.

According to still another embodiment, the present invention is a methodfor comparing motions, the method comprises: rendering in a computingdevice a first avatar from attributes derived from first motionperformed by a user, retrieving from a database a representation ofsecond motion performed by an instructor selected by the user from alist of authoritative instructors, showing on a display screen a displayof the first avatar next to the representation; and adjusting thedisplay in a perspective view determined by the user in responding to arequest from the user. Depending on implementation, the representationof the instructor may be an animated object or a second avatar renderedfrom the representation or the attributed from the second motion in datastore.

According to still another embodiment, the present invention is acomputing device for comparing motions, the computing device comprises:a processor, a transceiver coupled to the processor and receivingsensing data wirelessly from at least one sensor module in a pluralityof sensor modules disposed respectively and closely to designated bodyparts of a user, and a memory space coupled to the processor for storingcode. The code executed by the processor causes the computing device toperform operations of: rendering a first avatar from attributes derivedfrom first motion performed by the user, retrieving from a database arepresentation of second motion performed by an instructor selected bythe user from a list of authoritative instructors, showing on a displayscreen a display of the first avatar next to the representation, andadjusting the display in a perspective view determined by the user inresponding to a request from the user.

According to yet another embodiment, the present invention is an articleof clothing comprising: a layer of material, a plurality of sensormodules respectively attached to designated locations on the layer ofmaterial; and a plurality of conductive threads embedded in materials ofthe clothing, wherein one or more of the threads provide a communicationmedium between or among the sensor modules. In a preferable embodiment,the layer of material has an inner side, the sensor modules arerespectively attached to designated locations on the inner side of thelayer of material.

Other objects, features, benefits and advantages, together with theforegoing, are attained in the exercise of the invention in thefollowing description and resulting in the embodiment illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentinvention will become better understood with regard to the followingdescription, appended claims, and accompanying drawings where:

FIG. 1A shows an exemplary configuration in which a user inspecially-designed shirt and pants performs a pose or certain motionsaccording to one embodiment of the present invention;

FIG. 1B shows there are a number of sensor devices, sensor modules orsimply sensors placed respectively near certain human body parts;

FIG. 1C shows an exemplary layout of locations of the sensors on along-sleeve shirt;

FIG. 1D shows an exemplary layout of locations of the sensors on a pairof pants,

FIG. 2 shows a systemic functional block diagram according to oneembodiment of the present invention;

FIG. 3A shows an exemplary display a user may see, where a live avataris rendered and inserted into a display video;

FIG. 3B shows an example of a perspective view (e.g., a 45-Degree view)in which a user avatar is placed on the right side of a representationof a chosen teacher;

FIG. 3C shows an example of a direct overview, where a user avatar isplaced on the left side of a representation of a chosen teacher;

FIG. 3D shows a zoomed or close-up view of a user knee in a pose incomparison with the same performed by a chosen teacher, also in zoomedview;

FIG. 3E shows what is referred to herein as Live Pose Comparison (LPC)Mode;

FIG. 3F shows the pose comparison by comparing the bones (or frames) ofthe user or player avatar to the bones (or frames) of the referenceavatar;

FIG. 3G shows an exemplary score chart;

FIG. 3H shows that the length of all body segments can be approximatedas a ratio of user height (H);

FIG. 3I shows an example in which a user's hand is not rendered to touchthe floor;

FIG. 4A illustrates a quaternion of orientation q_(k) ^(bm) whichrotates the global reference frame {n} into the sensor (local) referenceframe {b} at time k;

FIG. 4B shows an algorithm block diagram implementing a linear Kalmanfilter to estimate the vertical direction in dynamic conditions by meansof the gyroscope and accelerometer data;

FIG. 4C shows an overall workflow of the three-dimensional orientationestimator according to one embodiment;

FIG. 4D shows the estimation of the bias b_(k) ^(b) given themeasurements gyri;

FIG. 4E illustrates an exemplary biomechanical model defined in thePIVOT Mag Free biomechanical protocol according to one embodiment of thepresent invention;

FIG. 4F shows that an N-pose assumption derives that each body segmentreference frame has the y-axis;

FIG. 4G shows (a) a grid search algorithm working principle for one bodysegment, angular grid, anatomical longitudinal axis, mediolateraldirection initial guess and refined mediolateral direction (black), and(b) a cost function over the grid for four body segments left and rightupper arm and left and right upper leg;

FIG. 4H shows each IMU has a global reference frame that depends on itsphysical orientation at the time in which a sensor fusion algorithm isstarted;

FIG. 4I shows a flowchart or process implemented in PIVOT Mag Freebiomechanical protocol;

FIG. 5A shows a flowchart or process of detecting stillness (no motion)in a user;

FIG. 5B shows the computations of attitude angles (phi, teta) for eachbody segment;

FIG. 5C shows a flowchart or process of performing attitude angleestimation;

FIG. 5D shows a flowchart or process of an exemplary reference poseclassifier;

FIG. 5E shows a flowchart or process of estimating user pose scoring;

FIG. 5F shows an exemplary sigmoid transfer function;

FIG. 6A shows a workflow or process of soft pose reset;

FIG. 6B shows a workflow or process of hard pose reset;

FIG. 7A shows a flowchart or process of double tap gesture detection;

FIG. 7B shows a flowchart or process of double tap event detection;

FIG. 7C shows an example of double tap event found on accelerometer dataafter applying two different peak thresholds;

FIG. 7D shows an example of double tap event found on gyroscope data;and

FIG. 7E shows a flowchart or process of double tap event detailsclassifier.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following detailed description of the present invention, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. However, it will become obviousto those skilled in the art that the present invention may be practicedwithout these specific details. In other instances, well known methods,procedures, components, and circuitry have not been described in detailto avoid unnecessarily obscuring aspects of the present invention.

The detailed descriptions of the present invention in the following arepresented largely in terms of procedures, steps, logic blocks,processing, and other symbolic representations that resemble dataprocessing devices capable of communicating with other devices. Thesedescriptions and representations are the means commonly used by thoseexperienced or skilled in the art to most effectively convey thesubstance of their work to others skilled in the art. The presentinvention includes one or more methods and systems for facilitating themanagement of motions. The methods along with systems including circuitsor architecture of computing devices to be described in detail below area self-consistent sequence of processes or steps leading to one or moredesired results. These steps or processes are those requiring physicalmanipulations of physical quantities. Usually, though not necessarily,these quantities may take the form of electrical signals capable ofbeing stored, transferred, combined, compared, displayed and otherwisemanipulated in a computer system or electronic computing devices. Itproves convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, operations,messages, terms, numbers, or the like. It should be borne in mind thatall of these similar terms are to be associated with the appropriatephysical quantities and are merely convenient labels applied to thesequantities. Unless specifically stated otherwise as apparent from thefollowing description, it is appreciated that throughout the presentinvention, discussions utilizing terms such as “processing” or “sending”or “verifying” or “displaying” or the like, refer to the actions andprocesses of a computing device that manipulates and transforms datarepresented as physical quantities within the computing device'sregisters and memories into other data similarly represented as physicalquantities within the computing device or other electronic devices.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment can be included in at least one embodiment of theinvention. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment, nor are separate or alternative embodiments mutuallyexclusive of other embodiments. Further, the order of blocks in processflowcharts or diagrams representing one or more embodiments of theinvention do not inherently indicate any particular order nor imply anylimitations in the invention.

One of embodiments in the present invention is to build a scalablecloud-based motion artificial intelligence (AI) platform (technology orsystem) with sensorized electronic garments to capture full-body motionsperformed by a wearer (user) in a 3-dimension space (3D) to givereal-time coaching feedback to learn proper motions or poses from aninstructor or teacher remotely located or in an application library (ordatabase). With the technology, human motions can be readily digitizedinto 3D data without using cameras, deployed to mass users for all kindsof creative 3D motion applications in sports, healthcare, AR/VR/gaming,and etc. For example, Yoga is an exemplary sport/exercise applicationthat may practice one embodiment of the present invention. One of theadvantages, benefits and objectives in the present invention is to helppeople to learn quickly how to move or pose properly in fitness, golf,tennis, soccer, dance, physical therapy rehabilitation, and etc.

Yoga will be used as an example or exemplary sport to facilitate thedescription of the present invention. A system providing Yoga employingone embodiment of the present invention is herein referred to as PIVOTYoga herein. According to one embodiment, PIVOT Yoga is a system, amethod, an apparatus or a part of system, wherein PIVOT Yoga includes atleast three elements. 1. An article of clothing or garment (a.k.a.,eGarment), worn by a yoga practitioner, is embedded with a plurality of(digital) sensors that observe and transmit the angles and relativepositions of body parts of the user in real time and 3D space to anexternal device (e.g., a smartphone). 2. A mobile application or App,executed in the external device, is designed to receive, process, andinterpret sensor data from the sensors, and displays a representation ofmotions by one of yoga teachers. 3. Within the App, embeddedintelligence is specific to each teacher, where the intelligence, alsoreferred to as Motion AI, is what tells a user to which adjustments tohis/her yoga pose are needed.

As used herein, any pronoun references to gender (e.g., he, him, she,her, etc.) are meant to be gender-neutral. Unless otherwise explicitlystated, the use of the pronoun “he”, “his” or “him” hereinafter is onlyfor administrative clarity and convenience. Additionally, any use of thesingular or the plural shall also be construed to refer to the plural orto the singular, respectively, as warranted by the context.

FIG. 1A shows an exemplary configuration in which a user 100 inspecially-designed shirt and pants 102 performs a pose or certainmotions according to one embodiment of the present invention. As furtherdescribed below, the shirt and pants 102 include a plurality of sensormodules (preferably not visible), each being affixed to a designatedlocation, preferably inside, of the clothing 102. With a designated Appexecuting in a portable device 104 (iPhone or wearable device), the user100 (i.e., a wearer of the clothing 102) can view on a display 106 howwell she is performing a pose (motion).

Subject to a preference, the user 100 may place such an exemplary device104 anywhere as long as it can maintain communication with the sensorsin the clothing 102. A display 106 may be shown from the device 104 oron a larger screen (e.g., via Chromcast). The user 100 may choose a yogaroutine from a list of activities in the App executed in the portabledevice 104, and then proceed with the routine. As will be furtherdetailed below, the user 100 may further choose an instructor or teacherfrom a list of available instructors to guide her exercise, where thechosen instructor may be asked for feedback for each pose or motion theuser 100 has just performed. The instructor, in her own voice, will thenverbally tell or show the user, for example, which body part to move, inwhich direction, and how far. In one embodiment, the portable device 104may provide verbal instructions from the chosen instructor or show avideo, where the user may control the video in various ways, e.g., voicecommand or taping on some body parts, and at any point during a pose,ask for comparison between the motions of herself and the choseninstructor.

FIG. 1B shows there are a number of sensor devices, sensor modules orsimply sensors 100 to be placed respectively near certain human bodyparts. According to one embodiment, these sensors are affixed torespective designated locations within the clothes 102 corresponding todesignated human body parts, for example, one sensor is responsible formonitoring a chest, another sensor is responsible for monitoring anupper arm, still another sensor is responsible for monitoring a flatarea just above an ankle. Depending on implementation, each of thesensors includes one or more inertial sensors that produce sensingsignals when the sensors are caused to move around with the wearer. Thesensing signals are sampled periodically (e.g., every 10 millisecond) toproduce sensing samples or data.

In one embodiment, the portable device 104 executing an App is caused toreceive or collect some or all the sensor samples from the sensors andtrack at every sample point if needed. A system is remotely located withrespect to but communicates with the portable device, wherein the systemis referred to as a server, a cloud computer or simply cloud, andconfigured or designed to perform motion analysis by processing a set ofraw sensor samples received remotely from one, more or all of thesensors (via the portable device), and derive joint angle outputs todetect start/end of motion, classify a motion type (e.g., forehandtopspin, backhand slice, flat serve, etc.) and compute importantattributes of the motion (e.g., speed, mass, distance, volume, velocity,acceleration, force, and displacement in scalar or vector). The bodysegment frames and motion analysis attributes are then sent to adesignated App (e.g., Yoga App) running in a mobile device, for 3Dgraphics rendering into a human avatar, animation and motion chartanalysis. Depending on implementation, some or all of the functions inthe system may be performed within the portable device 104.

FIG. 1B also shows two exemplary types of the sensor modules, asatellite module and a hub module. For ease of identification in oneembodiment, the hub module or hub is designed to have a unique shape,different from the rest of the satellite modules. The hub module is madein a distinct “+” medic shape in one embodiment. In some embodiments,the satellite modules connect wirelessly to a single Hub module, forexample, via Wi-Fi, Bluetooth, and etc. In one embodiment, the modulesmay communicate via a proprietary high-speed 2.4 GHz protocol. For somesport (e.g., tennis), the hub module may be typically disposed near thechest location and is configured to combine the sensor data with thesame timestamp and streams, for example, via Wi-Fi or Wi-Fi-Direct to acloud datacenter or a mobile device (phone, tablet, or laptop). Inanother embodiment, as will be further described below, the modules maycommunicate via a communication medium (e.g., conductive threadsembedded in an article of clothing).

In one embodiment, each of the satellite modules includes amicrocontroller, at least an inertial sensor and a transceiver forintercommunication with the hub module that includes a microcontroller,at least an inertial sensor and a transceiver for intercommunicationwith the satellite modules and another transceiver for communicatingwith an external computing device (e.g., the portable device). Each ofthe sensor modules produces sensing data at a predefined frequency whena user makes moves, the sensing data from the satellite modules arereceived in the hub module and combined with the sensing data generatedwithin the hub module and transported wirelessly to the external devicedesigned to derive the motion of the user performing activities andfacilitate a comparison between the derived motion with stored motion toillustrate a difference between the motion made by the user and motionmade by another person.

In another embodiment, each of the satellite modules includes aninertial sensor while the hub module includes a microcontroller, aninertial sensor and an interface for intercommunication with inertialsensors in the satellite modules and a transceiver for communicatingwith an external computing device (e.g., the portable device). Each ofthe inertial sensors produces sensing signals when a user makes moves,the sensing signals from the inertial sensors are received in the hubmodule via a communication medium (e.g., the conductive threads) andcombined with the sensing signal generated within the hub module. Thesensing signals are sampled at a predefined frequency and transportedwirelessly to the external device designed to derive the motion of theuser performing activities and facilitate a comparison between thederived motion with stored motion to illustrate a difference between themotion made by the user and motion made by another person.

According to one embodiment, an article of clothing, also referred toherein as sensorized eGarments (washable), body motions can be capturedand transmitted to an external device so that an authoritative teachermay be engaged to dynamically, in real-time, instruct a user how toimprove his motions, for nearly anything from sports to physicaltherapy. An exemplary sensor module may be, but not limited to, aninertial sensor, such an inertial sensor may be a 9-axis inertial sensorhaving accelerometer, gyroscope, and magnetometer, or a 6-axis inertialsensor having only accelerometer and gyroscope. Each sensor is placed ina specific location on the inner side of the garment to track the motionof every major limb (bone, body part or body segment). FIG. 1C shows anexemplary layout 116 of locations of the sensors on a long-sleeve shirt,these sensors are located specifically to capture the motion of aparticular body part (e.g., a finger or a forearm), where a hub sensoris to be disposed near the chest area of a human body when the shirt isworn. FIG. 1D shows an exemplary layout 120 of locations of the sensorson a pair of pants, these sensors are located specifically to capturethe motion of a particular body part (e.g., a knee or an ankle), where ahub sensor is to be disposed near the waist area of a human body.Depending on implementation, one or more batteries (e.g., buttonbatteries) are also embedded in the clothes.

According to one embodiment, a specially designed conductive thread 118or 122 is used in the clothing to provide connections between batteriesand the sensor modules if the batteries are not within each of thesensor modules, and between the hub module and satellite modules. Theconductive thread 118 or 122 has textile properties like a regular yarn,composed of low-resistivity (less than 1.5 Ohms per meter) copper corewith nano fiber insulation and capable of transmitting high speedelectrical signal (up to 10 Mbits per second). In one embodiment, thediameter of the conductive thread 118 or 122 is only 0.32 millimeters.In another embodiment, the conductive thread 118 or 122 goes a zigzagpattern to allow more stretches when needed. When worn, the eGarmentslook and feel like regular athletic-leisure clothes (athleisure) withthe electronics hidden and unfelt.

With the voice capabilities on the portable device, a user is able topause, resume, skip forward, freeze a video provided by the app. Forexample, a video or an avatar showing a perfect pose can be paused orrepeated, or viewed from different perspectives. The user may ask forfeedback while the video of an authoritative teacher is running.Depending on implementation, there are two ways to do this with voiceand/or gestures. Without using a wake word, a user, after a one-timesetup routine, can simply issue a command within earshot of his phone.The user can issue commands that the system pays attention to as thesystem is trained to recognize only his voice in one embodiment. As faras the gestures are concerned, since the clothes worn by the user aresensorized, the user may double-tap on various places on his body as away of controlling the app. In one embodiment, double-tapping on theleft hand pauses or resumes the video, double-tapping on the right handskips to the next chapter in the video, and double-tapping on the chestsensor asks the system for feedback. In another embodiment, a gesture isdesigned to freeze an avatar in a video. In still another embodiment,one of the sensors (e.g., the one on the waist) is designed to signal apause of the avatar or feedback of a chosen instructor.

FIG. 2 shows a systemic functional block diagram 200 according to oneembodiment of the present invention. A user wears a set of garmentsembedded with sensor modules 202. The number of sensor modules 202 andplacement can be determined depending on a target application and typesof motion to be captured and analyzed. According to one embodiment, eachsensor module comprises:

-   -   a 9-axis sensor chip having integrated 3-axis gyroscope, 3-axis        accelerometer, 3-axis magnetometer, such as those manufactured        by Invensense;    -   a 32-bit ARM Cortex M4F microcontroller (MCU) with floating        point arithmetic unit (FPU) to perform floating-point        math-intensive sensor fusion processing at every sensor module,        such as those manufactured by Freescale; and    -   a wireless chip with embedded 32-bit ARM Cortex M0 MCU to        support 2 Mbps wireless communication, such as a Nordic 2.4 GHz        wireless chip.

In one embodiment, the wireless chip is based on a proprietary andenhanced Shockburst protocol, which has been deployed formedical/industrial devices. Other standard wireless protocols likeBluetooth/BLE, Ant+ and ZigBee may also be employed. One of the sensormodules 202 is designed to function as a hub 204 of all the satellitesensor modules, controlling and collecting sensor data from thesatellite sensor modules. The sensor data from the satellite sensormodules are received and combined with the sensor data generated in thehub 204 into one record having the same timestamp and streamed out tothe cloud or the portable device. Typically, the sensor data samplingrate is at 100 Hz, producing gyro x/y/z, accel x/y/z, mag x/y/z, andquaternion w/x/y/z values for each satellite every 10 milliseconds. Toget robust data bandwidth and wireless distance to a Wi-Firouter/hotspot, the system may include a Wi-Fi module supporting802.11b/g/n. In the absence of Wi-Fi router/hotspot, the hub module canstream the sensor data directly to a mobile device 208 (e.g.,smartphone/tablet/laptop), for example, via Wi-Fi-Direct protocol. Ifthe mobile device 208 has limited computing resources compared to one ormore cloud servers 210, motion capture/analysis may be performed basedon reduced information from the sensor modules, but overall stilldelivering the benefits in the present invention.

In the presence of an Internet connection 206 to a cloud datacenter(e.g., the servers 210), the captured and combined sensor data recordsare streamed continuously to the cloud datacenter. The data streamqueuing and processing may use a framework suitable for real-time streamanalytics and having sub-second response time. In one embodiment, thesystem uses open-source software components, such as Kafka (for messagequeuing), Jetty (for application session management), and Rserve (forexecuting R math programs).

With a Kafka framework, the system can queue sensor data streaming fromthousands to millions of users, while maintaining low latencyrequirement for real-time processing. Multiple sensor records may bebatched to be processed by the known R math program. One or more Rprocesses may be dedicated for each user to compute the following: Jointangle estimate of each joint based on multi-sensor data and humanbiomechanics model, rotational direction values of corresponding bodysegments, detection of the start, middle, end, and type of a motion thatis unique to a target application, all based on a sequence ofmulti-sensor samples (called frames).

For example in tennis, a motion could be a forehand topspin with startframe at ready position, middle frame at ball contact, and end frame atcompletion of swing follow through. The motion is analyzed for differentattributes or statistics, such as (for tennis) number of repetitions,footwork quality metrics (number of steps before ball contact, knee bendangle, balance), power metrics (swing speed, hand acceleration, ballstrike zone), injury risk analysis (elbow, shoulder, wrist, back, knee),and etc., all based on the available joint angles, approximate rotationvalues of all 21 segments of human skeleton (wire body) that is ready tobe rendered and animated by a 3D graphics software like Unity(commercially available 3D game engine software).

To complete the streaming, the output of various (joint angle)processing and motion attributes/stats can be streamed out to a userassociated portable device to be further processed for live avataranimation and chart views. For playback and data analytics, every user'srecording session may be stored in a cloud database or in the portabledevice. Both the raw sensor data input and output results (e.g., jointangle frames, motion attributes/stats) can be part of the sessionrecord. For animation playback and chart views, the output data may beretrieved and sent to a mobile device. When there is enhancement oraddition to the motion capture and motion analysis algorithms, thesystem can re-generate the output results from the original input data.

The overall system stack comprises layers of hardware, firmware,wireless network, cloud infrastructure, real-time streaming software,biomechanics motion algorithms, database, big data analytics, 3Dgraphics, and a user interface. The following table summarizes thevarious aspects of the system.

Requirement System Feature 1. Capture and Employ inertial sensor chipsin smartphones and analyze human wearables to track movements. Multipleinertial motions with sensors may track movement of body segments. orwithout To get better sensor fusion and positioning accuracy, camera aprocessor is employed to integrate all 3 micro-electro-mechanical (MEM)accelerometer, gyroscope, and magnetometer. To further improve humanmotion capture and analysis, biomechanics modeling and knowledge of thetarget application's activities are combined. 2. Instant System performshigh-speed algorithms to analyze biomechanics motions based on humanbiomechanics model, joint feedback angles, raw sensor data, and sensorfusion. System incorporates proper biomechanics knowledgebase andpatterns into the motion library to compare with, based on the specificsof target application. These algorithms require substantial mathematicalcomputations. To give instant feedback in sub- second, the systemprovides a real-time stream processing in the cloud for scalablecomputing. 3. Injury The system may incorporate injury analysis andprevention motion library patterns/signatures based on analysis studiesin biomechanics, physical therapy/ rehabilitation, sports medicine, andexperiments in the target application area. The system may continuouslyadd more injury patterns into the motion library and algorithms andallow users to add their own injury patterns to recognize possibleinjury. 4. Live remote In one embodiment, the system leverages themotion cloud capabilities to enable real-time motion monitoring ormonitoring of a user by other authorized users coaching (coach, doctor,supervisor) from any distant places. Unlike video monitoring, the sensorstream bandwidth requirement may be several orders of magnitude less. 5.Share motion In one embodiment, the system leverages the cloudrecordings infrastructure to share motion recordings with other withauthorized users. authorized The system may record both the raw sensordata users input and output results (animation frames, motionattributes). When there is enhancement or addition to the motion captureand motion analysis algorithms, the system can re-generate the outputresults from the original input data. 6. Data The system may store alluser profiles and analytics recordings in the cloud's scalable database/insight storage. The system may deploy big data analytics tools andsearch queries to gain insight information upon request on user's owndata, or anonymous business intelligence. 7. Scalable and The systemplatform is based on an architecture adaptable with common buildingblocks that can scale and to many target adapt to customization and manytarget applications applications. In one embodiment, the systemleverages cloud's “infinite” computing resources to scale increasedapplication complexity, the number of active users, concurrent sessionsat peak usage, and newly developed applications. The system mayimplement on-demand cloud resource management with load balancing tohandle changing requirements and demands. 8. Affordable The system mayoptimize COGS (cost of goods sold) by choosing commodity/volume hardwarecomponents, cost-efficient contract manufacturers, and license-free opensource software packages. The system may optimize operational coststhrough on-demand cloud resources. 9. Easy to use The system may use anintuitive UI (user interface) with game technology where users of allages can operate easily without a manual or complex instructions. Thesystem may select features that give most benefits to users and presentthe feature capabilities in multi-level UI, starting from simple to deepanalysis. The system may use sensor packaging and harness designed forsimplicity and ease of use.

Referring now to FIG. 3A, it shows an exemplary display a user 300 maysee a live avatar 302 is inserted into a display video. During theplayback of an instructional video by a chosen teacher 304 in the PIVOTYoga App, there are some extra spaces on one side of a frame. Into theextra space, the live avatar 302 representing the user 300 may beinserted as a comparison to the chosen teacher 304. For example, as theuser raises his left arm, the corresponding avatar on the screen raisesits left arm as well. In one embodiment, the avatar is rendered based onthe sensor data received from the clothing embedded with the sensors. Inanother embodiment, one or more cameras are used to capture the pose ofthe user, images from the cameras are analyzed to derive a pose of theuser, where the derived pose is used to render the avatar. As differentcamera angles come into view, the avatar, or appropriate portion of theavatar in the event of a close-up, will automatically move into correctposition in view of the teacher.

Once in a pose, the user 300 may ask the system or strictly speaking,the chosen teacher for feedback on his pose. The request is received andrecognized (nearly instantaneously), the view on the display may change.Instead of being in a side-by-side video environment, the user is nowpresented in an environment that has been specially designed for posecomparison. It is herein to refer this environment as Live PoseComparison. According to one embodiment, the request may be generatedfrom one or more sensors by the user tapping on a specific part of hisbody or a voice from the user.

In one embodiment, the avatar representing the user is superimposed ontop of a reference avatar representing the teacher or a model designatedby the teacher. Directly to the side is a numbered diagram of the mat,each side of the mat presents a perspective view of the avatar-teachercombination, and the user may switch among those views by calling out aselected view with his voice.

FIG. 3B shows an example of a 45-Degree view, where the user avatar 302is placed on the right side of the chosen teacher 304. FIG. 3C shows anexample of a direct overview, where the user avatar 302 is placed on theleft side of the chosen teacher 304. FIG. 3D shows a zoomed or close-upview 320 of a user knee in a pose in comparison with the same performedby the chosen teacher 304, also in zoomed view.

FIG. 3E shows what is referred to herein as Live Pose Comparison (LPC)Mode. Instead of being in a side-by-side video environment, the user isnow presented in an environment that has been specially designed forpose comparison. In one embodiment, a user avatar 330 is superimposeddirectly on top of a reference avatar 332 (e.g., the teacher or adesignated model by the teacher). Directly to the side is a numbereddiagram of the mat, each side of the mat presents a different view ofthe avatar-teacher combination, and the user can switch among thoseviews by calling out with his voice. According to one embodiment, assoon as the LPC is displayed, a teacher voice announces feedback inresponse to the pose the user is holding at the moment the user askedfor feedback. This feedback may announce an important change that isneeded at the moment and all done in the teacher voice. For example, thevoice may indicate which body part needs to move, and how far in whichdirection.

In one embodiment, the pose comparison is done by comparing the bones(or frames) of the user or player avatar to the bones (or frames) of thereference avatar as shown in FIG. 3F. All of the bones are compared totheir corresponding counterparts (e.g. the right hand of the player iscompared to the right hand of the reference) using the basic 3D distanceformula. This gives a distance between the current poses of the playerand the reference. In one embodiment, the distance in Unity is measuredin a predefined unit (e.g., meters or inches). The Unity is the 3D gameengine software with which the system is designed.

With the results of this pose comparison, the player bone with thehighest distance to its direct counterpart is identified. Errors betweenthe two poses can then be determined. In one embodiment, an error isexpressed in 3 component vectors (X/Y/Z), and a largest error componentis to be corrected first. For example, if the bone with the largesterror is the right knee, and the largest component of the error is −0.2meters (negative corresponding to left) on the X Axis, then the playeris instructed to move his right knee 0.2 meters to the right. Thiscomparison is done in order and may be repeated if a threshold is notreached. In one embodiment, there is a default order. The player isinstructed to correct his feet first, then his legs, chest, hands, andarms (in that order).

In addition to the pose correction, the decision about what body part topresent for correction is a decision that can be made solely by theteacher. Each of the authoritative teachers adopted in the system mayspecify which errors in view of the pose differences to be corrected inany given pose, and a relative priority order of each of thesecorrections. In other words, different teachers may have differentprocedures to correct a pose error by a user (student). Accordingly, theselection of bones, the order in which they are corrected, and the axisthat has priority for each bone, are determined and configured for eachpose individually by a chosen teacher.

Regardless it is generic or teacher-specific pose correction, one of theimportant features is that the system (e.g., PIVOT Yoga) providesautomatic confirmation of a user's adjustment to a correction. In oneembodiment, as soon as the user had made the suggested correction(within a pre-set tolerance level), the App is designed to have aresponse from the teacher, e.g., “That's good!” or “Please move yourright arm a bit more right”.

In the LPC mode, the user avatar is superimposed onto the teacher avatar(normalized so that the heights of the two avatars are substantiallysimilar or the same), and the user has the control for changing whichside of his yoga mat is being displayed. If there are significant-enoughalignment problems on a particular side of a pose, that correspondingview is highlighted (e.g., in red). The assessment is based on atolerance level that may be predefined or set up by a user or theteacher.

A user may always rely on the avatar comparisons directly. The referenceavatar can be in a different color (e.g., yellow) and visually behindthe user avatar as shown in FIG. 3E, any yellow peeking through wouldgenerally indicate an alignment error for the user pose. Even though theuser avatar and reference avatar are displayed in 2-dimension (2D)planes, the raw avatar data (bones and joint angles) are being trackedand compared in full 3-dimension (3D) space accuracy. More viewingangles in any 2D views (e.g., top view) and any 3D views (e.g., rotatingin 360 or 720 degrees) can be provided and shown.

As an extension to the pose comparison, for each bone on the player, theaxis with the highest degree of error is identified and counted. Theaxis is used to determine which angle would give the player the bestview of his avatar for correcting his pose error. For example, if thereare 10 bones, the user receives correction messages 5X, 3Y, and 2Z. Inthis scenario, the user has the most errors in the X Axis (left/right),so top-down or frontal view may be selected based on other refiningfactors.

For the teacher-specific pose comparison, the system is designed toautomatically display to the user the camera view for the side of hispose which has the most severe alignment problems according to thechosen teacher. Based on the teacher's prioritized bone order ofcorrection, the camera angle is selected based on the prioritized bone'slargest error in the X, Y, or Z axis.

According to one embodiment, a user scoring algorithm is designed. Foreach pose, there is a 3D reference model (e.g., based on or from theteacher). Based on the model, it can be calculated how closely the useris approaching that pose in 3D. The difference may be reported, forexample, as a percentage. According to one embodiment, each frame isobserved while the user is nominally in a given pose. The frame that hasthe smallest percentage of overall deviation against the reference poseis selected with full 3D accuracy. The percentage is recorded as a scorefor that pose in the teacher's sequence on that day, and can be used totrack the pose (and display it to the user) over time. An underlyingscoring algorithm is to leverage the pose comparison. For each bone onthe player, the distance to its direct counterpart is saved. Thesedistances are compared against a set of thresholds to determine thescore. For example, if the minimum threshold is 0.05 meters, the maximumthreshold is 0.25 meters, and all bones are greater than 0.25 metersfrom their counterparts, then the player would receive a score of 0(zero). Likewise, if the bones were all less than 0.05 meters from theircounterparts, the player would receive a score of 100 (one-hundred).Values on this scale are determined by how far each bone is from itscounterpart. For a second example, if there are 10 bones, 8 of which arebelow the minimum threshold, and 2 of which are beyond the maximumthreshold, then the player would receive a score of 80%. FIG. 3G showsan example of scoring of poses performed by a player over a period,which also shows the score statistics of the player for each pose oraggregated poses of each session, tracking the progress over time andmultiple sessions. The App presents this scoring to show the user his“Best” or “Worst” pose, and enable the user to share each on a socialmedia, e.g., Facebook.

To give more realistic animation, the user avatar is modeled asaccurately as possible. In one embodiment, a user height is obtainedfrom the user. The height information is used for all calculations,especially for modeling the user avatar when to place the avatar on thescreen (e.g., where to place on the screen the head of the user who isbending at his knees). The standard anthropometric parameters are used,where the length of all body segments can be approximated as a ratio ofuser height (H) as shown in FIG. 3H. This approximation may differ incertain actual body segment length(s) of the user. In one embodiment,the user's actual body part lengths are adopted to scale the user avatarbody parts, through initial calibration poses that could be actual yogaposes.

There are times, particularly in Yoga, certain poses have well knownpositions in which known body parts must be on the ground and at certainjoint angles. For example, the Downward Facing Dog pose has both handsand feet on the ground (mat) with the legs, spine and arms fairlystraight. If a user is asked to be in this pose, and yet the useravatar's hands or feet are not planted on the ground/mat, certain bodysegment length(s) are scaled accordingly to match user's actual posewith hands and feet on the mat. To reduce variation of the pose,markings (e.g., hand prints, foot prints) on the mat are used toposition the user according to his height. So based on stored knowledgeof which poses require which, the length of the avatar's bones can bemathematically expanded or contracted. In general, a user can be askedto do multiple “calibration” poses to arrive to the best body segmentscaling of the avatar specific to the user.

Many calculations are made in real time to display all parts of a user'sbody properly. All of those calculations require an anchor which is asingle point that all measurements are based from. It turns out thatusing the same anchor for all poses can create some small discrepancies.After all, some poses in yoga are standing up; and some poses are lyingdown; and some poses are on the hands and knees. If the user is in apose with their hands on the floor, and we're using an anchor thatassumed the user was standing up, the result will be that the user'shands, in the motion capture display, would not actually seem to touchthe floor, as shown in FIG. 3I.

To address this, dynamic anchoring technique is designed to chooses ananchor point in the background based on a known pose. In one embodiment,the dynamic anchoring method ensures that any given avatar alwaysremains on the ‘ground’ in 3D space (in Unity). This is achieved byidentifying which bone on the avatar has the lowest position on the YAxis, and moving the entire avatar by the opposite of that value suchthat the lowest bone is always at ground level.

According to one embodiment, a motion capture system (referred herein asPIVOT Mag Free) is designed and relies on gyroscope and accelerometeronly. The inertial sensors in FIG. 1B are 6-axis inertial sensors havingonly gyroscopes and accelerometers. The design is based on an algorithmfor tilt estimation (angles with respect to the vertical direction)which is augmented with gyroscope integration for the heading estimation(angle about the vertical direction). A sensor fusion algorithm isdesigned to take the accelerometer and gyroscope measurements as aninput. The output is the quaternion of orientation q_(k) ^(bm) whichrotates the global reference frame {n} into the sensor (local) referenceframe {b} at time k, see FIG. 4A. The global reference frame of thesensor fusion algorithms running in each sensor has the z axis alignedwith the vertical direction and the x and y axis (in the horizontalplane) resulting from the initial sensor orientation.

In one embodiment, a linear Kalman filter is used in order to estimatethe vertical direction in dynamic conditions by means of the gyroscopeand accelerometer data. The algorithm block diagram is presented in FIG.4B. After the vertical direction is estimated, a horizontal referenceneeds to be available in order to compute the tree dimensionalorientation of the sensor reference frame with respect to the globalreference frame. Since a magnetometer is not used in PIVOT Mag Free,this horizontal direction can be computed with the gyroscope data. Attime zero, the horizontal reference measured in the body frame isassumed to be a known canonical vector, i.e., the x axis (1, 0, 0) ofthe global reference frame. As the time passes and the body moves, thegyroscope data is used to project in time the horizontal reference inthe body reference frame. The time-propagation equation of a 3D vector,given the body angular velocities, is given by the following equation:

$\{ \begin{matrix}{h_{0}^{b} = ( {1,0,0} )} \\{h_{k + 1}^{b} = {{- {\exp( {\lbrack \omega_{k}^{b} \rbrack T_{s}} )}}h_{k}^{b}}}\end{matrix} $

Once the horizontal and vertical directions are known in the globalreference frame (known a priori) and in the sensor reference frame(estimated with sensor data), the orientation of the global referenceframe with respect to the sensor reference frame can be computed. In oneembodiment, the TRIAD method being a very popular computationalprocedure is used. FIG. 4C shows an overall workflow of thethree-dimensional orientation estimator according to one embodiment.

A non-null bias in the gyro measurements sometimes results in a driftingerror in the output of the angular speed time integration. Such a biascan be accurately estimated by averaging gyroscope measurements whilethe sensors are still (e.g., on a table). However, the accuracy of theestimated bias is heavily degraded by possible user motions during thecalibration procedure. For this reason, the Kalman filter is designedfor being able to estimate the gyroscope biases while the sensors areworn by the user. The rationale is to use a more complex algorithm (aKalman filter vs. a simple average) to deal with user's (moderate tolow) motion during the gyroscope calibration.

In one embodiment, the gyroscope measurements gyr_(k) ^(b) are modeledas the sum of the true angular velocity ω_(k) ^(b), the gyroscope biasb_(k) ^(b) and the white noise v_(k) ^(b):gyr _(k) ^(b)=ω_(k) ^(b) +b _(k) ^(b) +v _(k) ^(b)The aim of the Kalman filter in PIVOT Mag Free is the estimation of thebias b_(k) ^(b) given the measurements gyri, as shown in FIG. 4D. Thebias and the angular velocity are considered as the inner state of adynamical system. These two states are modeled as time-constantvariable. These states are also considered to be the output of thedynamical system, as shown in FIG. 4D. They can be derived from thegyroscope measurements as follows:

$\{ \begin{matrix}{{\hat{b}}_{k}^{b} = {\sum\limits_{i = 1}^{N}{gyr}_{k}^{b}}} \\{{\hat{\omega}}_{k}^{b} = {{gyr}_{k}^{b} - {\hat{b}}_{k}^{b}}}\end{matrix} $

A biomechanical protocol implemented in PIVOT Mag Free includes a set ofdefinitions and computational procedures that allow relating sensorreadings (angular velocity and acceleration) and sensor fusion output(quaternions) to the motion of the body segment. Three main blocks arerequired to reach this goal: the biomechanical model definition, theSensor-To-Segment (STS) calibration and the alignment of the globalreference frames of each sensor attached to the body.

Biomechanical Model

The biomechanical model defined in the PIVOT Mag Free biomechanicalprotocol is shown in FIG. 4E according to one embodiment of the presentinvention. Specifically, FIG. 4E shows an anatomical reference frame forall of the 16 body segments considered in PIVOT Mag Free. The y-axis isassumed to be the longitudinal axis for all the body segments while thex-axis is assumed to coincide with the medio-lateral direction (positivefrom left to right). Therefore, the z-axis is represented by theantero-posterior direction (positive from the front to the back). In oneembodiment, fourteen body segments are considered: six segments for theupper limb: upper arm, forearm and hand (left and right), six segmentsfor the lower limb: thigh, shank and foot (left and right), trunk andpelvis.

Sensor-to-Segment Calibration

It is known to those skilled in the art that there are two mainapproaches: anatomical methods, where a user is asked to stay still insimple and known body poses (N-pose, T-pose and etc.) and functionalmethods, where the user is asked to perform very simple calibrationmotions. In the former, the estimated quaternions are compared with theones expected in the known static pose. From this comparison, the STSmisalignment (STS quaternion) is estimated with the aim of compensatingthe IMUs quaternions during the motion capture session, where IMU standsfor inertial measurement unit, each IMU is included in a sensor module.In the latter, body rotational axes are measured through the calibrationmotion in the sensor reference frame and then used to build the STSorientation.

As described above, the quaternions returned by PIVOT Mag-Free sensorfusion running for each body segment refer to a different globalreference frame. In one embodiment, a functional method (two-stepprocedure) is used to estimate the STS quaternion based on the rawsensor data, i.e., accelerometer and gyroscope data.

Vertical Direction Estimation: N-Pose

The first step of the proposed calibration method includes asking a userto stay still in N-pose. For each sensor, accelerometer data is measuredand averaged during this time window to produce an estimate of thereaction to the gravity (vertical direction, upward) seen from thesensor reference frame. The N-pose assumption derives that each bodysegment reference frame has the y-axis (see FIG. 4E and FIG. 4F) whichis aligned with the vertical direction. For this reason, the averagedaccelerometer vector represents an estimate of the longitudinaldirection of the relative anatomical reference frame.

Medio-Lateral Direction Estimation: Functional Motions

The second step of the STS calibration implemented in PIVOT Mag Free isrepresented by the functional motions. In this stage, the user, startingfrom an N-pose, is asked to perform the following motion sequence asshown in FIG. 4F:

-   -   90 degrees flection of the upper arms in the sagittal plane,        keeping the forearm and hand rigid with respect to the upper        arms.    -   90 degrees flection of the left thigh in the sagittal plane,        keeping the shank and the foot rigid with respect to the thigh;        and    -   90 degrees flection of the right thigh in the sagittal plane,        keeping the shank and the foot rigid with respect to the thigh.

During each motion, gyroscope measurements are acquired and normalizedto one. The circular average of the normalized gyroscope represents theestimate of the Medio-lateral direction of that body segment seen fromthe corresponding sensor reference frame. No functional motions arerequired for the trunk and pelvis because sensor position on those bodysegments is quite constrained by the garment design. Therefore, themedio-lateral direction is assigned a-priori according to the physicalorientation of the sensor with respect to the body.

Medio-Lateral Direction Refinement: Angular Grid-Search Method

The medio-lateral direction estimated during the functional motionsdescribed above can have a poor inter-subject repeatability. This isespecially true for those body segments with relevant number of softtissues and muscles such as the upper arms and the thighs. For thisreason, in PIVOT Mag Free a computational procedure was introduced whichis called angular grid-search. The underlying idea of this method is toconsider the output of the functional motions output (i.e., the averagemean of the gyroscope data) as an initial guess which is then refinedbased on the computation of a cost function over a pool of candidatedirections. In the following, the main steps of the grid-searchalgorithm are listed and detailed.

Angular Grid Computation

The angular grid is represented by a pool of directions which lie in thehorizontal plane and are equally distributed in the range of +/−60degrees with respect the initial guess. The horizontal plane is computedas the plane orthogonal to the vertical direction estimated during thefirst N-pose by means of the accelerometer data. In FIG. 4G, the line440 represents the vertical direction while the lines 442 represent theguess directions. Such grid is built by computing all the directions inthe horizontal plane with an increment of about 1.5 degrees, spanningthe +/−60 degrees range with respect to the initial guess. The initialguess is represented by the projection onto the horizontal plane of theaverage gyroscope vector computed during the functional motions (theline 444). The refined (and final) direction is represented by the line446.

Cost Function Computation

The cost function is based on the assumptions that functional motionsare performed within the sagittal plane with null pronation. Hence, forthe arm segments (upper arms, forearms and hands) the cost function tobe minimized is represented by the horizontal adduction during the armfunctional calibration (stage 2 in FIG. 4F). Same rule is applied to theleg segments, considering the corresponding functional calibrationstage. The direction which results in the cost function minimum is takenas the medio lateral direction for that segment. As an example, in FIG.4G, the cost function for the left upper arm, right upper arm, leftupper leg and right upper leg are reported.

Final STS Orientation Estimation: TRIAD Method

From the biomechanical model definition, the vertical and medio-lateraldirections of all body segments are assigned as (0, 1, 0) and (1, 0, 0)respectively. On the other hand, both directions have been measured inthe two-step procedure described above. Therefore, given this coupledvector observations, the STS orientation can be computed easily by meansof the TRIAD method.

Global Reference Frame Alignment

Each IMU has a global reference frame that depends on its physicalorientation at the time in which the sensor fusion algorithm is started,as shown in FIG. 4H. Hence, a further compensation is needed when jointangles are to be computed. It should be noted that this additional stepwould not be required by standard sensor fusion algorithms that make useof magnetometer measurements. In fact, it is possible to define a uniqueglobal reference frame for all the sensors when magnetometer data areused. For this reason, an additional computational stage is added in thePIVOT Mag Free biomechanical protocol workflow. This procedure needs tobe repeated every time a new motion capture session is started (i.e.,all the sensor fusion algorithms are started on all the body segments).Two assumptions are necessary: the user is in an N-pose and STScalibration quaternions are already available.

For example, considering the trunk and the right upper arm, thefollowing equation holds:q ^(gRUAgTRK)=(q ^(sRUAgRUA))−1⊗(q ^(bRUAsRUA))−1⊗q ^(bRUAbTRK) ⊗q^(bTRKsTRK) ⊗q ^(sTRKgTRK)If assumption 1 hold, then q^(bRUAbTRK)=(1,0,0,0), since the trunk andright upper arm body segments are aligned during N-Pose:q ^(gRUAgTRK)=(q ^(sRUAgRUA)|_(Npose))⁻¹⊗(q ^(bRUAsRUA))−1⊗q ^(bTRKsTRK)⊗q ^(sTRKgTRK)|_(Npose)If assumption 2 holds, then q^(bRUAsRUA) and q^(bTRKsTRK) are known andthe global reference frame misalignment q^(gRUAgTRK) can be computed.

In PIVOT Mag Free, the trunk is taken as the global reference.Therefore, the generic formula to estimate the global reference framemisalignments between the reference (trunk) and any other body segment Xcan be computed as follows:q ^(gXgTRK)=(q ^(sXgX)|_(Npose))⁻¹⊗(q ^(bXsX))⁻¹ ⊗q ^(bTRKsTRK) ⊗q^(sTRKgTRK)|_(Npose)It shall be noted that this estimated global reference misalignment willhold for all the motion capture sessions, but it needs to be recomputedif the sensor fusion algorithms are restarted.

The joint orientation computation between two body segments is computedwith the following formula:q ^(bDISbPRX) =q ^(bDISsDIS) ⊗q ^(sDISgDIS) ⊗q ^(gDISgTRK)⊗(q^(gPRXgTRK))⁻¹⊗(q ^(sPRXgPRX))⁻¹⊗(q ^(bPRXsPRX))⁻¹where DIS stands for distal and PRX stand for proximal. This formularepresents the way all the blocks described above (i.e., sensor fusionoutputs, sensor to segment calibration and global alignments) are puttogether to estimate the joint orientations. It is also shown how thetrunk global reference frame is taken as reference for all the othersegments. The joint quaternion will then be transformed into jointangles with standard quaternion to Euler angles formula.

FIG. 4I shows a flowchart or process 460 implemented in PIVOT Mag Freebiomechanical protocol. The very first phase is represented by the STScalibration, which consists in the first N-pose plus the functionalmotions. Then, to start a new motion capture session, sensor fusionalgorithms are started. Since the user is supposed to stay in N-pose, itis possible to compute the global reference frame alignment describedabove. After that is accomplished, the real motion capture can start,with the joint angles computation as described in the formula above.Every time that the motion capture needs to be restarted, the user needsto go back to the global alignment stage to recompute the globalreference frame alignment.

As the background application scenario involves the chance of anorientation drift, the yoga poses detection and classification algorithmmust rely on measures unaffected by such errors. In one embodiment, ayoga pose is approximated with a quasi-static condition lasting for morethan a second, it is then possible to exploit accelerometers data tocompute body segments' attitude, thus inferring user's current pose inreal time. A detection algorithm, also referred to herein as TuringSensePose Detection Algorithm (TSPDA) is composed by the followingsub-components or steps:

-   -   1) a model of each pose is detected, the model can be based on        unprocessed accelerometers data or attitude angles, where the        model may also comprise specific tolerance bounds for every body        segment;    -   2) an algorithm capable of detecting user's absence of        significant motion over a pre-defined time window in real-time;    -   3) an algorithm capable of extracting body-segments attitude        angles from raw accelerometers' data, when the user is        classified as ‘still’ or ‘not moving’ by the algorithm mentioned        in (2);    -   4) a classifier capable of comparing body-segments attitude        angles computed in (3) with the pose-models mentioned in (1),        the classifier identifies in real time if the user is assuming        one of the poses defined in the model databased (1). Even if no        pose is detected, the classifier will detect a pose closer to        the user's body configuration. The classifier is designed to be        capable of improving the classification using the current 3D        joint angles computed by the Mag Free algorithm;    -   5) a specific scoring system to provide the feedback on how much        the user was close to the pose detected on (4), where the        scoring system algorithm provides:        -   a bounded (i.e., 0%-100%) score describing the overall user            ability to match the reference pose        -   a bounded (i.e., 0%-100%) score describing, for every body            segment, the user's ability to match the reference pose.

Every pose to be detected will be modelled and described, based on thedata collected with the TS Mag-Free system, in terms of: body segmentsattitude, body segments-specific attitude tolerances, body segments rawaccelerations, body segments-specific raw accelerations tolerances, 3Djoint angles, and joint-specific 3d joint angles tolerances andstatistical weights.

An algorithm, also referred to as Body-Motion Detection Algorithm, isdesigned to collect and update in real time a rolling buffer ofgyroscopes' data; the length of such buffer is pre-defined (e.g., 1second), but can be changed at any time during the recording. Data iscollected from the sensors placed on or near designated body segments ofa user. Once the analysis buffer is filled (e.g., after the first secondof recording), gyroscopes data is postprocessed, averaged and comparedagainst pre-defined thresholds. If the current postprocessed andaveraged gyroscopes data coming from all involved body segments is foundlower than the pre-defined thresholds, the user's state shall beclassified as “not moving”. Only if the user's state is classified as“not-moving”, the pose detection and classification will proceed furtherwith the above steps (3) (4) and (5). FIG. 5A shows a flowchart orprocess 500 of detecting stillness (no motion) in a user.

An algorithm, also referred to as Body Segments' Attitude EstimationAlgorithm, is designed to collect and update in real time a rollingaccelerometers data buffer of a pre-defined length (i.e., 1 second). Thedata is collected from the Sensors placed on all user's body segmentsinvolved by the TS Mag Free protocol. In the same fashion, if alreadypresent, 3D joint angles data can be collected as well to improve theclassification. Once the analysis buffer is filled (i.e., after thefirst second of recording), accelerometers data will be postprocessedand averaged.

Knowing the sensors orientation a-priori, accelerations can be expressedin the body-segment system of reference, the attitude angles (phi, teta)for each body segment are computed as shown in FIG. 5B:

${{phi} = {a\tan( \frac{a_{x}}{\sqrt{a_{y}^{2} + a_{z}^{2}}} )}}{{teta} = {a{\tan( \frac{a_{y}}{\sqrt{a_{x}^{2} + a_{z}^{2}}} )}}}$Attitude angles (phi, teta) for each body segment are stored and usednext in the above sub-components (4) (5) for the pose classification andscoring. If present, 3D joint angles are stored as well. FIG. 5C shows aflowchart or process of performing Attitude Angle Estimation.

All parameters computed in step (3) are compared with the models for allthe poses described in step (1). In order to compute the degree ofmatching between one user's body segment and the reference valuecontained in one reference pose model (1), separate methods are applied:M1:

-   -   a) divide user's body segment averaged 3d acceleration by its        Euclidean norm |a|=norm(a);    -   b) compute the angle between the 3d vector |a| and the 3d vector        contained in the reference pose (|aref|)        -   delta_angle=a cos(|a| DOT |aref|)        -   where DOT is the dot product.            M2 2:

a) compute the difference between the body segments' attitude angles(teta, phi) calculated in (3) and the corresponding values saved in thereference pose model (1). M3: similarly to M1 1, if available, computethe overall degree of the agreement between the user's joint angles andthe values saved in the reference pose model.

Once (M1), (M2), and eventually (M3), are computed for each bodysegment, their values are combined into a per-pose cost function whichwill assign an overall agreement score among the user's current pose andthe pose model used for the comparison. Once all pose models (1)agreement scores have been computed, the model pose with the highestagreement score is selected as the user's current pose. FIG. 5D shows aflowchart or process 540 of Reference Pose Classifier.

Once the current user's pose is classified (4) and matched with a posemodel in the reference pool (1), it is possible to compute the followingoutcomes:

-   -   O1: current user's pose MATCHES or NOT MATCHES the reference        pose model;    -   O2: overall pose matching score; and    -   O3: per-body segment matching score.        Comparison metrics computed in (4) are combined in a segment-        and pose-specific cost function representing the degree of        matching between the user's current, actual pose and the        reference.

In the following diagram is described the classification of every userbody segment into:

1) matching/not matching the reference pose model (Cost_score−O1, O2)

-   -   cost_score=1 indicates that the body segment matches the        reference model    -   cost_score=0 indicates that the body segment does not match the        reference model

2) Overall percentage of agreement with the reference pose model (PA%−O2)

Once the current user's pose is classified in step (4) and matched witha pose model in the reference pool (1), it is possible to compute thefollowing outcomes as shown in a flowchart or process 550 FIG. 5E.Depending on the method applied, the per-segment and per-pose costfunction can be unbounded and ranging from minus-infinity andplus-infinity. In order to normalize the score and make it at the sametime comparable inter-user and inter-session and easily interpretable, asegment-specific and pose-specific sigmoid transfer function is appliedto the cost function in order to translate the original unbounded valuesinto bounded and easy-readable ones (e.g., 0%-100%). The sigmoidtransfer function can be calibrated on the user specific level ofability in performing the pre-defined pool of poses (i.e., novice,amateur, expert, professional), so that the final score can be adaptedto his experience progression.

As described above, one of the strategies to deal with the angulardrifting error in PIVOT Mag Free is the pose reset. The idea is toexploit those moments while the user is doing a yoga pose (and it isstill) to restart the sensor fusion algorithms. Note that the userstillness alone is not a sufficient condition to apply a sensor fusionreset in the mag free scenario. In fact, after the reset, each sensorfusion algorithm will take a different global reference frame.Therefore, the same global reference frame alignment procedure explainedabove needs to be performed. For doing so, it is necessary that thephysical orientation of the sensor is known at the time the sensorfusion algorithm is reset. There are two reset strategies possible andhereafter they are called as soft pose reset and hard pose reset.

FIG. 6A shows a workflow or process 600 of soft pose reset. The process600 may be launched every time a user is detected to be still. In thattime instant, the current orientation of all the body segments of theuser is stored and a sensor fusion algorithm reset is issued. Thequaternions produced after the sensor fusion reset will be matched withthe stored orientations before the reset in order to compute theprevious global reference frame alignment procedure as described above.After the compensation, the motion capture session can be resumed.

The process 600 takes advantage in accuracy through the sensor fusionreset. However, it must be noted that the resuming condition for thesensor fusion algorithms could already be affected by some drift.Repeating this procedure many times may still result in a slowaccumulation of drifting errors. In fact, the expectation of the softpose reset is to make the drift slower but not to produce drift-lessmotion capture. Despite this drawback, however, this procedure isrelatively simple to be implemented.

Another reset solution is implemented in PIVOT Mag Free in order toprovide a drift-less estimate over a longer time window, which is calledhard pose reset. This procedure is very similar to the soft posereference, as shown in FIG. 6B with the following two main differences:

-   -   1. The reset must be triggered when the user is in a specific        yoga pose (N-pose or yoga pose), the yoga pose detector        described above is therefore used to trigger the hard pose        reset.    -   2. The global reference frame alignment is computed based on        reference (pre-recorded) body segment orientations. These        reference poses can also be acquired with other (more accurate)        motion capture systems, like stereo-photogrammetric systems.

The advantage of the hard pose reset over the soft pose reset is thatnot only sensor fusion algorithms are reset, but the reference pose usedto resume the mocap is pre-recorded. This means that the reference poseis drift free, maybe even acquired with high accuracy systems. For thisreason, triggering this procedure multiple times during will result indrift-less motion capture. However, this procedure is more complex andrequires a specific yoga pose detection algorithm on top of it.

The description of the present invention is now focusing on what isreferred herein as Double Tap Gesture Detection. The purpose of doubletap gesture detection is to detect in real time a user performingspecific gestures while wearing smart-clothing. Detected gestures areused to trigger specific actions on the App as instance play, and topause the video. For double tap, the technique is intended to detect theact of tapping twice with one of the user hands' palm or fingers overone body segment. It is also possible to detect double tapping gesturesover objects such as desks or walls. At least one involved body segment(the “active” or “tapping” hand and the “passive” or “tapped”) issupposed to be instrumented with a smart-clothing.

The technique is based on the analysis of accelerometers and/orgyroscopes data coming from MEMS (micro electro mechanical systems) orNEMS (nano electro mechanical systems) contained within the smartclothing. In order to allow the detection of the broadest possiblespectrum of combinations of tapping locations, data from all availablesensors locations will be acquired and processed. This will allow todetect any combination of double-tapping events, both being performedwith an instrumented body segment over another (T2), or with oneinstrumented body segment over a non-instrumented one (T1). Examples ofa tapping event happening between two instrumented body segments:

-   -   T2.1: instrumented left hand double-tapping on instrumented        right hand    -   T2.2: instrumented left hand double-tapping on instrumented        chest    -   T2.3: instrumented right hand double-tapping on instrumented        left forearm Examples of a tapping event happening between one        instrumented body segment and a non-instrumented body        segment/object:    -   T1.1: instrumented left hand double-tapping on non-instrumented        right hand    -   T1.2: non-instrumented left hand double-tapping on instrumented        chest    -   T1.3: instrumented left hand double tapping on the desk    -   T1.4: instrumented right hand double tapping on the wall        It is possible to detect only double taps in which the time        intercurring the two tapping events is lower than a predefined        threshold; such threshold can be altered while the algorithm is        already running.

The technique is composed by a chain of 4 specific functional blocks:

-   -   FB1: sensor data collection and buffering;    -   FB2: sensor data processing;    -   FB3: Double-Tap event detection on a single sensor; and    -   FB4: Double-Tap events aggregator and classifier.        FIG. 7A shows a flowchart or process 700 of double tap gesture        detection. Data from all sensors locations (e.g., left hand,        right hand, chest) is collected and streamed to the detection        algorithm using one rolling data buffer (FIFO) per sensor, per        location. The buffer will contain as much data as the maximum        allowed duration of the double tap. As instance, considering:    -   100 Hz sampled data; and    -   a maximum allowed time of 1 second for the second tap to happen.

The technique is composed by a chain of 4 specific functional blocks.The rolling buffer contains 100 (100 Hz*1 sec) samples per satellitelocation. For every sensor location, the algorithm will be aware of thenumber and type of sensors present:

-   -   S1: Accelerometer    -   S2: Gyroscope    -   S3: Accelerometer AND Gyroscope        Depending on the scenario (S1-S3), the algorithm will apply        different, specific signal processing methods (e.g., high-pass        or low-pass filtering) in order to remove spurious noise and to        isolate only the signals' spectra band required. Additional        postprocessing methods can be applied in order to maximize the        chances of double tap detection; in the following examples:    -   Gyroscopes data will be processed to compute the frame-by-frame        norm; norm is the frame-by-frame Euclidean norm of the gyroscope        data.

(f1)|g|=√{square root over (g _(x) ² +g _(y) ² +g _(z) ²)}

-   -   Accelerometers data will be processed to compute the normalized        jerk; normalized jerk is the Euclidean norm of the        time-derivative of the acceleration data.

$\begin{matrix}{\underline{j} = \frac{{da\_}(t)}{d(t)}} & ({f2a})\end{matrix}$ $\begin{matrix}{{❘j❘} = \sqrt{j_{x}^{2} + j_{y}^{2} + j_{z}^{2}}} & ({f2b})\end{matrix}$

After processing the incoming signal, an iterative, dynamic thresholdapproach will be applied on the buffered and processed data. Aftersetting the amplitude threshold value, a local maxima detectionalgorithm will seek the buffered data for signal peaks above the setthreshold. A local maximum is identified by the following rules:

-   -   R1: three subsequent frames above the set threshold    -   R2: data at frame t has a lower amplitude than data at frame t+1    -   R3: data at frame t+1 has a higher amplitude than data at frame        t+2        Considering the number of above-threshold local maxima found,        the signal can be classified as:    -   E1: no local maxima are found higher than the threshold;    -   E2: 1 local maximum is found higher than the threshold;    -   E3: 2 local maxima are found higher than the threshold; and    -   E4: more than 2 local maxima are found higher than the        threshold.        If 2 or more local maxima are found (E3, E4), the iterative        process will stop. If 2 local maxima are found (E3), a double        tap event occurring in the current location is identified.        If 1 or none local maxima are found (E1, E2), the amplitude        threshold value is lowered by a predefined value (e.g., 5%) and        the local maxima algorithm is iterated again. This method is        iterated until one of the following events occur:    -   E3: 2 local maxima are found;    -   E5: the amplitude threshold reached the lowest value allowed        (e.g., 200 deg/sec).        FIG. 7B shows a flowchart or process 710 Double Tap Event        Detection Flowchart. FIG. 7C shows an example of double tap        event found on accelerometer data after applying two different        peak thresholds. FIG. 7D shows an example of double tap event        found on gyroscope data.

Based on the events occurred (E1-E4), a first classification of thesignal takes place on the sensor data:

-   -   E1: NO DOUBLE TAPPING occurred, as no related signal's pattern        landmarks are found    -   E2: NO DOUBLE TAPPING occurred, as no related signal's pattern        landmarks are found    -   E3: DOUBLE TAPPING EVENT OCCURRED    -   E4: NO DOUBLE TAPPING occurred, as the signal's pattern found is        related to motion noise or non-conforming gestures (e.g., triple        taps, hands shaking).        If a double tap event is found on 2 or more sensors locations,        the data from the sensors which generated the events is passed        to the active/passive segment classifier.        The active/passive segment classifier will analyze the data and        provide:    -   O1: if present, which sensor location was the “active” tapping        segment (or “tapper”)    -   O2: if present, which sensor location was the “passive” tapping        segment (or “tapped”).

If more than two sensors locations generated the double tap event (E3),as instance due to a particular fast user motion, the classifier will becapable of detecting the actual 01 and 02 by analyzing and comparingsignal patterns characteristics from all sensors locations whichgenerated the double tap event (E3). FIG. 7E shows a flowchart orprocess 750 of double tap event details classifier. Once thedouble-tapping event is detected and classified in all of its parameters(time, O1, O2), the algorithm communicates the decoded event to the App,which applies the specified action.

As instance:

Example 1

-   -   Tapping Time: 0.4 seconds;    -   O1: left hand is the active tapper;    -   O2: right hand is the passive tapper; and    -   This combination of events is associated to the app's command:        “Stop video”.

Example 2

-   -   Tapping time: 0.5 seconds;    -   O1: right hand is the active tapper;    -   O2: no events; and    -   This combination of events is associated to the app's command        “Resume Video”.

The App or the algorithms described above are preferably implemented insoftware, but can also be implemented in hardware or a combination ofhardware and software. The implementation of the App or the algorithmscan also be embodied as computer readable code on a computer readablemedium. The computer readable medium is any data storage device that canstore data which can thereafter be read by a processor or a computersystem. Examples of the computer readable medium include read-onlymemory, random-access memory, CD-ROMs, DVDs, magnetic tape, optical datastorage devices, and carrier waves. The computer readable medium canalso be distributed over network-coupled computer systems so that thecomputer readable code is stored and executed in a distributed fashion.

The present invention has been described in sufficient details with acertain degree of particularity. It is understood to those skilled inthe art that the present disclosure of embodiments has been made by wayof examples only and that numerous changes in the arrangement andcombination of parts may be resorted without departing from the spiritand scope of the invention as claimed. Accordingly, the scope of thepresent invention is defined by the appended claims rather than theforegoing description of embodiments.

We claim:
 1. A system for capturing motion performed by a user, thesystem comprising: a plurality of sensor modules respectively attachedto different body parts of a user, each of the sensor modules generatinga sensing signal when the user makes moves, the each of the sensormodules including a microcontroller, at least an inertial sensor and atransceiver for communicating wirelessly with a computing deviceremotely located with respect to the sensor modules, wherein a set ofsensing data is produced from the sensing signal from the each of thesensor modules at a predefined frequency, the sensing data from at leastone of the sensor modules are wirelessly received in the computingdevice to derive motion of the user performing the moves and render anavatar of the user moving in real time based on the derived motion. 2.The system as recited in claim 1, wherein the sensing data from the atleast one of the sensor modules are processed and incorporated into anapplication running on the computing device for rendering animation ofthe avatar based on the derived motion.
 3. The system as recited inclaim 2, wherein joint angle outputs are derived from combining thesensing data from the at least one of the sensor modules to detect astart and an end of the motion, and classify the derived motion to apredefined type.
 4. The system as recited in claim 3, wherein attributesof the derived motion made at each of the different body parts of theuser are derived from the combined sensing data.
 5. The system asrecited in claim 4, wherein the sensing data from each of the sensormodules is synchronized in time with each other.
 6. The system asrecited in claim 4, wherein the attributes of the derived motion areincorporated into 3D graphics rendering the animation.
 7. The system asrecited in claim 6, wherein the attributes of the derived motion includeone or more of speed, mass, distance, volume, velocity, acceleration,force, and displacement in scalar or vector.
 8. The system as recited inclaim 6, wherein the animation is shown on a display device.
 9. Thesystem as recited in claim 6, wherein the animation is shown in a chosenperspective while the user performs various moves.
 10. A method forcapturing motion performed by a user, the method comprising: generatinga plurality of sensing signals from a plurality of sensor modulesrespectively attached to different body parts of a user, each of thesensor modules including a microcontroller, at least an inertial sensorand a transceiver for communicating wirelessly with a computing deviceremotely located with respect to the sensor modules, producing a set ofsensing data from each of the sensing signal at a predefined frequency;and transporting wirelessly a plurality of sets of sensing data to thecomputing device to derive motion of the user performing the moves andrender an avatar of the user moving in real time based on the derivedmotion.
 11. The method as recited in claim 10, further comprising:processing the sets of sensing data; incorporating the processed sensingdata into an application running on the computing device to renderanimation of the avatar based on the derived motion.
 12. The method asrecited in claim 11, wherein said processing the sets of sensing datacomprises: deriving joint angle outputs from the sensing data from atleast one of the sensor modules to detect a start and an end of themotion, and classifying the motion to a predefined type.
 13. The methodas recited in claim 12, wherein said processing the sets of sensing datacomprises: deriving attributes of the derived motion made at each of thedifferent body parts of the user.
 14. The system as recited in claim 13,wherein the sets of the sensing data from each of the sensor modules aresynchronized in time with each other.
 15. The system as recited in claim13, wherein said incorporating the processed sensing data into anapplication comprises incorporating the attributes of the derived motioninto 3D graphics rendering the animation.
 16. The system as recited inclaim 13, wherein the attributes of the derived motion include one ormore of speed, mass, distance, volume, velocity, acceleration, force,and displacement in scalar or vector.
 17. The system as recited in claim11, wherein the animation is shown on a display device.
 18. The systemas recited in claim 17, wherein the animation is shown in a chosenperspective while the user performs various moves.